A wide image with a diagram of branching in git.

A tibble (sort of a data frame)

library("tidyverse")
library("gapminder")
library("kableExtra")
gapminder |> slice_sample(n=5) |>  kable(format="html") |> kable_styling(font_size=10)

• the symbol |> or |> is the so-called pipe operator: it inputs what’s on its left to what’s on its right

• One would obtain the same result by typing slice_sample(.data=gapminder,n=5)

dplyr verbs

In a tibble, observations are on rows and variables are on columns

filter

The filter verb just retrieves observations that meet one or more conditions.

Say we want the data for European countries after year 2000.

gapminder |> 
  filter(continent=="Europe",year>=2000) |> slice(1:6) |>
  kable(format="html") |> kable_styling(font_size=10)

filter

filter(condition1,condition2) returns the observations that meet condition1 AND condition2

• For more complex conditions, one can use the logical operators

gapminder |> 
  filter((year>=2000)|(lifeExp>70)) |>  slice(1:6) |> 
  kable(format="html") |> kable_styling(font_size=10)

filtered_gapminder = gapminder |> 
  filter((year>=2000)|(lifeExp>70)) 

arrange

• The arrange verb re-orders (ascending) the observations according to a variable.

gapminder |> 
  arrange(gdpPercap) |>  slice(1:4) |> 
  kable(format="html") |> kable_styling(font_size=8)

gapminder |> 
  arrange(gdpPercap) |>  slice(1:4) |> arrange(country) |>
  kable(format="html") |> kable_styling(font_size=8)

arrange

• To arrange in descending order, it just takes

gapminder |> 
  arrange(desc(gdpPercap)) |>  slice(1:4) |> 
  kable(format="html") |> kable_styling(font_size=8)

slice

The slice verb picks the observations up according to their position in the tibble

slice1=gapminder |>  slice(1:3) |> 
  kable(format="html") |> kable_styling(font_size=8)

We can indicate some specific positions

slice2=gapminder |>  slice(c(20,37,49)) |> 
  kable(format="html") |>
  kable_styling(font_size=10)

Or we can pick them up at random, using slice_sample(n= )}$

slice3 = gapminder |>  slice_sample(n=3) |> 
  kable(format="html") |>
  kable_styling(font_size=10)

slice_min(n= ) and slice_max(n= ) are combinations of arrange and slice. Check ?slice_min for help

select

The select verb refers to variables, indicated by names

gapminder |>  
  select(country,gdpPercap) |> 
  slice(1:4) |> kable(format="html") |>  kable_styling(font_size=10)

gapminder |>  
  select(country:lifeExp) |> 
  slice(1:4) |> kable(format="html") |>  kable_styling(font_size=10)

select: helper functions

Selecting variables by name becomes increasingly tedious as then nunber of variables to deal with increases.

mutate

The mutate verb can modify and/or create new variables

• One may want to express the population in millions, and create a variable with the full Gdp, not just the Gdp per-capita

gapminder |> 
  mutate(pop=round(pop/1000000,2),
         gdp=gdpPercap*pop) |> 
  select(pop, contains("gdp")) |> 
  slice_sample(n=3) |> 
  kable(format="html") |> kable_styling(font_size = 10)

summarize

The summarize makes it very easy to compute descriptive stats of given variable

gapminder |> 
  summarize(min_gdp=min(gdpPercap),
            q1_gdp=quantile(gdpPercap,.25),
            median_gdp=quantile(gdpPercap,.5),
            mean_gdp_pc=mean(gdpPercap),
            q3_gdp=quantile(gdpPercap,.75),
            max_gdp=max(gdpPercap)
            )  |> kbl() |> kable_styling(font_size=12)

group by

The group_by verb imposes a conditioning on the further operations. It works great with summarize, to have conditional descriptive statistics

gapminder |> 
  group_by(continent) |> 
  summarize(min_gdp=min(gdpPercap),
            q1_gdp=quantile(gdpPercap,.25),
            median_gdp=quantile(gdpPercap,.5),
            mean_gdp_pc=mean(gdpPercap),
            q3_gdp=quantile(gdpPercap,.75),
            max_gdp=max(gdpPercap)
            )  |> kbl() |> kable_styling(font_size=12)

group by

The group_by verb imposes a conditioning on the further operations. It works great with summarize, to have conditional descriptive statistics

gapminder |> 
  filter(year>2000) |> 
  group_by(year,continent) |> 
  summarize(min_gdp=min(gdpPercap),
            q1_gdp=quantile(gdpPercap,.25),
            median_gdp=quantile(gdpPercap,.5),
            mean_gdp_pc=mean(gdpPercap),
            q3_gdp=quantile(gdpPercap,.75),
            max_gdp=max(gdpPercap)
            ) |> kbl() |> kable_styling(font_size = 8)

distinct

gapminder |> 
  distinct(year) |> kbl() |> kable_styling(font_size = 10) 

gapminder |> 
  distinct(year, continent) |> nrow() |> 
  kbl() |> kable_styling(font_size = 10) 

count

gapminder |>
  filter(year==2007) |> 
  count(continent) |> kbl() |> kable_styling(font_size = 10) 

gapminder |> 
  filter(year==2007) |> 
  mutate(high_low_lexp = ifelse(lifeExp>65,"high","low"))  |> 
  count(continent,high_low_lexp) |> 
  kbl() |> kable_styling(font_size = 10) 

count vs group_by + summarise

gapminder |> 
  filter(year==2007) |> 
  mutate(high_low_lexp = ifelse(lifeExp>65,"high","low"))  |> 
  group_by(continent,high_low_lexp) |>
  summarize(n=n()) |> 
  kbl() |> kable_styling(font_size = 10)

gapminder |> 
  filter(year==2007) |> 
  mutate(high_low_lexp = ifelse(lifeExp>65,"high","low"))  |> 
  count(continent,high_low_lexp) |> 
  kbl() |> kable_styling(font_size = 10) 

count vs group_by + summarize

Same as before, this time compute the relative frequencies, too :::{.columns} ::::{.column}

gapminder |> 
  filter(year==2007) |> 
  mutate(high_low_lexp = ifelse(lifeExp>65,"high","low"))  |> 
  group_by(continent,high_low_lexp) |>
  summarize(n=n()) |> 
  mutate(relative_freqs= round(n/sum(n),3)) |> 
  kbl() |> kable_styling(font_size = 10)

::::

gapminder |> 
  filter(year==2007) |> 
  mutate(high_low_lexp = ifelse(lifeExp>65,"high","low"))  |> 
  count(continent,high_low_lexp) |>
  mutate(relative_freqs=round(n/sum(n),3)) |> 
  kbl() |> kable_styling(font_size = 10) 

:::

count vs group_by + summarize

There is clearly something wrong with group_by + summarize, as the relative frequencies do not add up to one

count vs group_by + summarize

Same as before, this time compute the relative frequencies, too :::{.columns}

gapminder |> 
  filter(year==2007) |> 
  mutate(high_low_lexp = ifelse(lifeExp>65,"high","low"))  |> 
  count(continent,high_low_lexp) |>
  mutate(relative_freqs=round(n/sum(n),3)) |> 
  kbl() |> kable_styling(font_size = 10) 

gapminder |> 
  filter(year==2007) |> 
  mutate(high_low_lexp = ifelse(lifeExp>65,"high","low"))  |> 
  group_by(continent,high_low_lexp) |>
  summarize(n=n()) |> 
  ungroup() |> 
  mutate(relative_freqs= round(n/sum(n),3)) |> 
  kbl() |> kable_styling(font_size = 10)

:::

what is data taming?

import data

The first step in analysis is to import the data from another platform, or in another format - readr is the package providing functions to import common format, e.g. csv or txt - readxls and haven provide import functions for a variety of file types xls, sql, json

data taming tools

Some packages are particularly useful for data taming

• cast types: readr import functions (e.g. read_csv) have an option to specify the variable types as the data is imported (col_types)

tidy data

• Each row corresponds to a different observation

• Each column corresponds to a different variable

• Each cell corresponds to a unique combination observation/variable

Consider four students, and register whether they passed some of the test during a course.

tidify a table

To make this table to be tidy, we need a single column recording whether the test is passed or not

untidy_tab_1 |> pivot_longer(names_to = "test_name", 
                              values_to = "passed?",
                              cols=homework_1:final_proj_3) |> 
  kbl(format="html") |> kable_styling(font_size=8)

tidify a table

To make this table to be tidy, we need a single column recording whether the test is passed or not

untidy_tab_2 |> pivot_longer(names_to = "student_name", 
                              values_to = "passed?",cols=A:D) |> 
  arrange(student_name) |> select(student_name,everything()) |> 
  kbl(format="html") |> kable_styling(font_size=8)

tidify a table

To make this table to be tidy, we need different columns for different variables

tidy_tab=long_tab |> pivot_wider(names_from = variable_name, 
                              values_from = value) 
  
tidy_tab |> kbl(format="html") |> kable_styling(font_size=8)

tidify a table

glimpse(tidy_tab) 

Rows: 4
Columns: 4
$ student      <chr> "A", "B", "C", "D"
$ presences    <chr> "11", "10", "11", "10"
$ mode         <chr> "live", "live", "online", "live"
$ tests_passed <chr> "2", "3", "2", "0"

Note: presences and tests_passed are coded as character.

tidy_tab=tidy_tab |> mutate(across(.cols=c(presences,tests_passed), ~parse_double(.)))
glimpse(tidy_tab) 

Rows: 4
Columns: 4
$ student      <chr> "A", "B", "C", "D"
$ presences    <dbl> 11, 10, 11, 10
$ mode         <chr> "live", "live", "online", "live"
$ tests_passed <dbl> 2, 3, 2, 0

ggplot2: the grammar of graphics in R

• In ggplot2, graphics are made of different layers

• the mapping operation assigns a variable in a tibble to an element of a plot: different plots may have different elements to map:aesthetics refer to axes (x and y), but also to color, size.

pengs=palmerpenguins::penguins |> na.omit()
pengs |> ggplot(mapping=aes(x=flipper_length_mm,y=bill_length_mm,size=body_mass_g,color=island))

Nothing happens! (we just created the base layer)

ggplot2: the grammar of graphics in R

Depending on what we want to display, we can add a geom to the layer: to create a scatterplot we need to add points

pengs |> ggplot(mapping=aes(x=flipper_length_mm,y=bill_length_mm,size=body_mass_g,color=island))+
  geom_point(alpha=.5)

ggplot2: the grammar of graphics in R

The geom choice depends on the nature of the mapped variables: barplots are for factors

pengs |> ggplot(mapping=aes(x=species,fill=island))+
  geom_bar()

ggplot2: the grammar of graphics in R

For unordered factors, one usually wants to display the bars according to their occurrence.

• This is easily done using fct_infreq}$ and fct_rev}$ functions from the forcats}$ package

pengs |> ggplot(mapping=aes(x=fct_infreq(species),fill=island))+
  geom_bar()

pengs |> ggplot(mapping=aes(x=fct_rev(fct_infreq(species)),fill=island))+
  geom_bar()

ggplot2: the grammar of graphics in R

For single distributions, one may want to use an histogram, if one wants the relative frequencies instead of counts, the y axis has to be specified.

pengs |> ggplot(mapping=aes(x=body_mass_g))+
  geom_histogram(aes(y=..density..),bins =15,fill="indianred",color="darkgrey",alpha=.3)+
  geom_density(fill="cyan",alpha=.25)

ggplot2: the grammar of graphics in R

More advanced plot are easily obtained

pengs |> ggplot(mapping=aes(x=flipper_length_mm,y=bill_length_mm,size=body_mass_g,color=island))+
  geom_point(alpha=.5)+facet_grid(sex~species)